{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Image Classification Inference using TF Lite Runtime\n",
    "In this example notebook, we describe how to use a pre-trained Classification model for inference using the ***Tensorflow Lite Runtime*** interface.\n",
    "   - The user can choose the model (see section titled *Choosing a Pre-Compiled Model*)\n",
    "   - The models used in this example were trained on the ***ImageNet*** dataset because it is a widely used dataset developed for training and benchmarking image classification AI models. \n",
    "   - We perform inference on a few sample images.\n",
    "   - We also describe the input preprocessing and output postprocessing steps, demonstrate how to collect various benchmarking statistics and how to visualize the data.\n",
    "   \n",
    "## Choosing a Pre-Compiled Model\n",
    "We provide a set of precompiled artifacts to use with this notebook that will appear as a drop-down list once the first code cell is executed.\n",
    "\n",
    "<img src=docs/images/drop_down.PNG width=\"400\">\n",
    "\n",
    "## Image classification\n",
    "Image classification is a popular computer vision algorithm used in applications such as, object recongnition, traffic sign recongnition and traffic light recongnition. Image classification models are also used as feature extractors for other tasks such as object detection and semantic segmentation.\n",
    "   - The image below shows classification results on few sample images.\n",
    "   - Note: in this example, we used models trained with ***ImageNet*** because it is a widely used dataset developed for training and benchmarking image classifcation AI models\n",
    "\n",
    "<img src=docs/images/CLS.PNG width=\"500\">\n",
    "\n",
    "\n",
    "## Tensorflow Lite Runtime based Work flow\n",
    "The diagram below describes the steps for Tensorflow Lite Runtime based workflow. \n",
    "\n",
    "Note:\n",
    "- The user needs to compile models(sub-graph creation and quantization) on a PC to generate model artifacts.\n",
    "    - For this notebook we use pre-compiled models artifacts\n",
    "- The generated artifacts can then be used to run inference on the target.\n",
    "- Users can run this notebook as-is, only action required is to select a model. \n",
    "\n",
    "<img src=docs/images/tflrt_work_flow_2.png width=\"400\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import cv2\n",
    "import numpy as np\n",
    "import ipywidgets as widgets\n",
    "from scripts.utils import get_eval_configs\n",
    "last_artifacts_id = selected_model_id.value if \"selected_model_id\" in locals() else None\n",
    "prebuilt_configs, selected_model_id = get_eval_configs('classification','tflitert', num_quant_bits = 8, last_artifacts_id = last_artifacts_id)\n",
    "display(selected_model_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'Selected Model: {selected_model_id.label}')\n",
    "config = prebuilt_configs[selected_model_id.value]\n",
    "config['session'].set_param('model_id', selected_model_id.value)\n",
    "config['session'].start()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define utility function to preprocess input images\n",
    "\n",
    "Below, we define a utility function to preprocess images for the model. This function takes a path as input, loads the image and preprocesses the images as required by the model. The steps below are shown as a reference (no user action required):\n",
    "\n",
    " 1. Load image\n",
    " 2. Convert BGR image to RGB\n",
    " 3. Scale image\n",
    " 4. Apply per-channel pixel scaling and mean subtraction\n",
    " 5. Convert RGB Image to BGR. \n",
    " 6. Convert the image to NCHW format\n",
    "\n",
    "\n",
    "- The input arguments of this utility function is selected automatically by this notebook based on the model selected in the drop-down"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess(image_path, size, mean, scale, layout, reverse_channels):\n",
    "    # Step 1\n",
    "    img = cv2.imread(image_path)\n",
    "    \n",
    "    # Step 2\n",
    "    img = img[:,:,::-1]\n",
    "    \n",
    "    # Step 3\n",
    "    img = cv2.resize(img, (size, size), interpolation=cv2.INTER_CUBIC)\n",
    "     \n",
    "    # Step 4\n",
    "    if mean is not None and scale is not None:\n",
    "        img = img.astype('float32')\n",
    "        for mean, scale, ch in zip(mean, scale, range(img.shape[2])):\n",
    "            img[:,:,ch] = ((img.astype('float32')[:,:,ch] - mean) * scale)\n",
    "    # Step 5\n",
    "    if reverse_channels:\n",
    "        img = img[:,:,::-1]\n",
    "        \n",
    "    # Step 6\n",
    "    if layout == 'NCHW':\n",
    "        img = np.expand_dims(np.transpose(img, (2,0,1)),axis=0)\n",
    "    else:\n",
    "        img = np.expand_dims(img,axis=0)\n",
    "    \n",
    "    return img"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create the model using the stored artifacts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tflite_runtime.interpreter as tflite\n",
    "\n",
    "tflite_model_path = config['session'].get_param('model_file')\n",
    "artifacts_dir = config['session'].get_param('artifacts_folder')\n",
    "\n",
    "tidl_delegate = [tflite.load_delegate('libtidl_tfl_delegate.so', {'artifacts_folder': artifacts_dir})]\n",
    "\n",
    "interpreter = tflite.Interpreter(model_path=tflite_model_path, experimental_delegates=tidl_delegate)\n",
    "interpreter.allocate_tensors()\n",
    "\n",
    "input_details = interpreter.get_input_details()\n",
    "output_details = interpreter.get_output_details()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run the model for inference\n",
    "\n",
    "### Preprocessing and Inference\n",
    "\n",
    "   - You can use a portion of images provided in `/sample-images` directory to evaluate the classification inferences. In the cell below, we use a loop to preprocess the selected images, and provide them as the input to the network.\n",
    "\n",
    "### Postprocessing and Visualization\n",
    "\n",
    " - Once the inference results are available, we postpocess the results and visualize the inferred classes for each of the input images.\n",
    " - Classification models return the results as a list of `numpy.ndarray`, containing one element which is an array with `shape` = `(1,1000)` and `dtype` = `'float32'`, where each element represents the activation for a particular ***ImageNet*** class. The results from the these inferences above are postprocessed using `argsort()` to get the `TOP-5` class IDs and the corresponding names using `imagenet_class_to_name()`.\n",
    " - Then, in this notebook, we use *matplotlib* to plot the original images and the corresponding results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from scripts.utils import get_preproc_props\n",
    "\n",
    "# use results from the past inferences\n",
    "images = [\n",
    "    ('sample-images/elephant.bmp', 221),\n",
    "    ('sample-images/laptop.bmp', 222),\n",
    "    ('sample-images/bus.bmp', 223),\n",
    "    ('sample-images/zebra.bmp', 224),\n",
    "]\n",
    "size, mean, scale, layout, reverse_channels = get_preproc_props(config)    \n",
    "print(f'Image size: {size}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tqdm\n",
    "import matplotlib.pyplot as plt\n",
    "from scripts.utils import imagenet_class_to_name\n",
    "\n",
    "plt.figure(figsize=(20,10))\n",
    "\n",
    "for num in tqdm.trange(len(images)):\n",
    "    image_file, grid = images[num]\n",
    "    img = cv2.imread(image_file)[:,:,::-1]\n",
    "    ax = plt.subplot(grid)\n",
    "\n",
    "    img_in = preprocess(image_file , size, mean, scale, layout, reverse_channels) \n",
    "    if not input_details[0]['dtype'] == np.float32:\n",
    "        img_in = np.uint8(img_in)\n",
    "\n",
    "    interpreter.set_tensor(input_details[0]['index'], img_in)\n",
    "    interpreter.invoke()\n",
    "    res = interpreter.get_tensor(output_details[0]['index'])\n",
    "\n",
    "    # get the TOP-5 class IDs by argsort()\n",
    "    # and use utility function to get names\n",
    "    output = res.squeeze()[1:]\n",
    "    classes = output.argsort()[-5:][::-1]\n",
    "    outputoffset = 0 if(output.shape[0] == 1000) else 1 \n",
    "    names = [imagenet_class_to_name(x+outputoffset)[0] for x in classes]\n",
    "  \n",
    "    # plot the TOP-5 class names\n",
    "    ax.text(20, 0 * img.shape[0] / 15, names[0], {'color': 'red',  'fontsize': 18, 'ha': 'left', 'va': 'top'})\n",
    "    ax.text(20, 1 * img.shape[0] / 15, names[1], {'color': 'blue', 'fontsize': 14, 'ha': 'left', 'va': 'top'})\n",
    "    ax.text(20, 2 * img.shape[0] / 15, names[2], {'color': 'blue', 'fontsize': 14, 'ha': 'left', 'va': 'top'})\n",
    "    ax.text(20, 3 * img.shape[0] / 15, names[3], {'color': 'blue', 'fontsize': 14, 'ha': 'left', 'va': 'top'})\n",
    "    ax.text(20, 4 * img.shape[0] / 15, names[4], {'color': 'blue', 'fontsize': 14, 'ha': 'left', 'va': 'top'})\n",
    "    \n",
    "    # Show the original image\n",
    "    ax.imshow(img)\n",
    "    \n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot Inference benchmarking statistics\n",
    "\n",
    " - During the model execution several benchmarking statistics such as timestamps at different checkpoints, DDR bandwidth are collected and stored. `get_TI_benchmark_data()` can be used to collect these statistics. This function returns a dictionary of `annotations` and the corresponding markers.\n",
    " - We provide the utility function plot_TI_benchmark_data to visualize these benchmark KPIs\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Note:</b> The values represented by <i>Inferences Per Second</i> and <i>Inference Time Per Image</i> uses the total time taken by the inference except the time taken for copying inputs and outputs. In a performance oriented system, these operations can be bypassed by writing the data directly into shared memory and performing on-the-fly input / output normalization.\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from scripts.utils import plot_TI_performance_data, plot_TI_DDRBW_data, get_benchmark_output, print_soc_info\n",
    "stats = interpreter.get_TI_benchmark_data()\n",
    "fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(10,5))\n",
    "plot_TI_performance_data(stats, axis=ax)\n",
    "plt.show()\n",
    "\n",
    "tt, st, rb, wb = get_benchmark_output(stats)\n",
    "print_soc_info()\n",
    "\n",
    "print(f'{selected_model_id.label} :')\n",
    "print(f' Inferences Per Second    : {1000.0/tt :7.2f} fps')\n",
    "print(f' Inference Time Per Image : {tt :7.2f} ms')\n",
    "print(f' DDR usage Per Image      : {rb+ wb : 7.2f} MB')"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
